Repository navigation
ENH: Replace the shell wheel build system with Python (supersedes #302) - #309
hjmjohnson wants to merge 131 commits into
Conversation
d2ec481 to
00357a9
Compare
dzenanz
left a comment
There was a problem hiding this comment.
I have taken a look. There is too much code to look carefully. I tried running locally:
M:\a\IPP>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
Error: × could not find pixi.toml or pyproject.toml with tool.pixi at directory M:\a\IPP
M:\a\IPP>cd C:\Misc\ITKPythonPackage
M:\a\IPP>c:
C:\Misc\ITKPythonPackage>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
WARN Encountered 2 warnings while parsing the manifest:
⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
81 │
82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
83 │ │ linux = "5.4"
84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
· ╰──── declare these on the `platforms` entries instead
85 │
╰────
help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]
⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
94 │ platforms = ["osx-arm64"]
95 │ system-requirements = { macos = "14.0" }
· ─────────┬────────
· ╰── declare these on the `platforms` entries instead
96 │
╰────
help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]
Error launching 'python3': The system cannot find the path specified. (os error 3)
C:\Misc\ITKPythonPackage>pixi run python "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
WARN Encountered 2 warnings while parsing the manifest:
⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
81 │
82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
83 │ │ linux = "5.4"
84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
· ╰──── declare these on the `platforms` entries instead
85 │
╰────
help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]
⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
94 │ platforms = ["osx-arm64"]
95 │ system-requirements = { macos = "14.0" }
· ─────────┬────────
· ╰── declare these on the `platforms` entries instead
96 │
╰────
help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]
Running >>>>>: git fetch --tags origin ; # in cwd=C:\Dev\ITK-git with check=False
Traceback (most recent call last):
File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 628, in <module>
build_wheels_main()
~~~~~~~~~~~~~~~~~^^
File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 450, in build_wheels_main
raise RuntimeError(
...<2 lines>...
)
RuntimeError: Failed to fetch ITK tags in C:\Dev\ITK-git: fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
C:\Misc\ITKPythonPackage>
|
This seems to be doing something: |
|
That failed too: run.log |
ef0c2d4 to
1af2b15
Compare
Windows verification complete — handing back to Linux/macOSEverything Windows-specific on this branch is now verified by real builds, and one cross-platform defect found along the way is fixed. Head is What was verified on Windows 11 / VS 2022 (i7-9800X, 32 GB)
Defects fixed on this branch by the Windows workAll Windows-only unless noted; none were reachable from Linux/macOS testing.
Plus three test fixes so the suite runs on Windows ( For Linux/macOS
|
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
c219d94 to
afdfaf8
Compare
|
Thanks for running it locally — your Your build failure is our bug, not your invocation. The log ends at:
This PR stays WIP until the Doxygen prerequisite is resolved. "Too much Platform verification status as of afdfaf8
97 unit tests pass; One known caveat: a single aarch64 clean build failed while linking with a |
afdfaf8 to
e53db8d
Compare
|
@dzenanz thank you for the log file — it identified a real bug, and that bug is This should have been a What your log found, and the fixYour run failed at:
Two changes:
Six regression tests cover both (103 total, up from 97). Your exact command |
fb3e45b to
42922e0
Compare
1af2b15 to
ca9994a
Compare
ctypes structure fields are Any to mypy; the Windows memory probe now converts ullTotalPhys to int before dividing.
Thirteen findings, none a behavior change. Four guards tested a CMakeArgumentBuilder for truth, which is always true since it has no __len__; they only skipped an update() that is a no-op when empty, so they are removed. The remote-module path goes through required_module_source_dir(), and paths added to command lists are converted with str(), as echo_check_call did for them already.
Bare dict, list, Callable, CompletedProcess, Match, and PathLike get their arguments. CompletedProcess[str] shows echo_check_call's bytes branches were dead: run_commandLine_subprocess always runs with text=True, so stdout and stderr are printed directly.
parameter_option now indexes its defaults, so an unknown option raises KeyError where it used to return None into a string multiplication. push_dir declares its attributes, which mypy had read as always None.
test_non_git_source_is_not_fetched and test_git_source_still_fetches installed a fake git and never called build_wheels, so they passed whatever the guard did. The fetch-and-checkout block moves into sync_itk_source(), which the tests now drive through each outcome: no repository, a clean sync, a fatal and a survivable fetch failure, and a failed checkout. The source-text checks for the guard, the gitauto default, and the cache-tree error message become calls to the code.
The definitions every ITK wrapping build is configured with move from the builder's constructor into itk_source_build_configuration(), so a test can read them without the CMake and pixi paths the constructor resolves. The check for ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES calls it instead of searching the source text.
The check searched wheel_packaging.py for the exclusion flag. It now builds a tree holding an ITK .git directory, runs the real tar command create_posix_tarball produces, and confirms the archive keeps the sources and holds no .git entry at all. Excluding only the directory's contents fails it.
mypy's disallow_untyped_calls reported 61 calls into unannotated test helpers and one into build_tarball(). The helpers now carry the builder classes, Path, and str types they are actually used with; _make_builder is generic over the builder class so each fixture keeps its concrete type. build_tarball() returns None on every platform. Any appears only in the Path.open stand-in in test_build_jobs.py: it forwards whatever positional and keyword arguments the code under test passes to open(), which are genuinely dynamic.
mypy inferred the recording dict in test_install_step_runs_cmake_install_into_the_prefix as holding lists only, from its first assignment, and so judged the `check` comparison against True impossible. The dict holds a command list and a bool; declaring it dict[str, object] states that, and the comparison is checked as written.
Three tests reached the standard library through the module under test (build_jobs.platform, build_jobs.os, pyproject_configure.sys), which those modules never export. Each name is the same module object as the test's own import, so patching platform, os, or sys directly replaces exactly what the code under test sees.
Every flag strict enables now passes on its own, so the seven per-flag lines give way to strict = true, which is what sp-repo-review MY101 asks for. Unannotated functions stay allowed, the staged path the Scientific Python style guide describes: disallow_untyped_defs and disallow_incomplete_defs are off, and check_untyped_defs still checks their bodies. warn_unreachable and the three extra error codes are not part of strict and stay.
A contributor without pixi needs one entry point to the checks CI runs. noxfile.py wraps the existing commands rather than defining new ones: lint runs pre-commit on every file, tests runs pytest with the pyproject.toml settings against the dev dependency group on Python 3.11 to 3.14, and docs runs the sphinx-build line from the docs workflow (off by default; --serve uses sphinx-autobuild). The file follows the Scientific Python tasks guide: a uv script block and shebang, a minimum nox version, uv with a virtualenv fallback, and default= per session. Wheel builds stay pixi tasks because nox does not manage compiled builds well.
Contributors need to know which sessions exist, which run by default, how to pass arguments through, and that wheel builds remain pixi tasks.
compiler_cache.py picks the tool by platform and returns the launcher definitions. ITK_USE_CCACHE is set explicitly: ITK turns it on whenever a launcher is set and then runs CastXML and SWIG through the launcher, which ccache caches and sccache rejects. The cache is sized at 30G unless the caller set CCACHE_MAXSIZE or SCCACHE_CACHE_SIZE, since a wrapped ITK build overflows ccache's 5 GB default.
The flag was combined with the environment by "or", so --no-use-ccache could not turn off a cache that USE_CCACHE=1 asked for. An explicit flag now wins and the environment decides only without one. The flag gains the name --use-compiler-cache, since on Windows it means sccache; --use-ccache stays as an alias, and USE_COMPILER_CACHE joins USE_CCACHE.
oneTBB, and ITK when the superbuild builds it, now compile through the same cache. ITK_USE_CCACHE is forwarded with it, so the superbuild's ITK does not default it on and put SWIG and CastXML behind sccache.
ccache 4.14 for the macOS, Linux, and manylinux builds, sccache 0.19 for Windows. The lock adds them and their dependencies (libhiredis, xxhash) and changes nothing else.
The container mounts a host directory as its CCACHE_DIR, one per manylinux version and architecture, since ccache keys on the compiler and the x64 runner and the root aarch64 container must not share ownership of one directory. ITK_CONTAINER_CCACHE_DIR overrides the location. USE_CCACHE passes through, and the ITK wheel build no longer forces --no-use-ccache, so the host decides.
When a cache is used, its statistics are zeroed before the build steps and shown after them, even when a step fails, so each daily build logs its own hit rate. The sccache server keeps the environment it started with, so it is restarted after the cache size is set and stopped at the end. The report runs unchecked and cannot mask a build failure.
Daily releases rebuild mostly unchanged sources on every platform. Each native release build passes --use-compiler-cache, and the manylinux builds pass USE_CCACHE=1 into the container, which keeps its cache in a host directory between runs. A cache hit returns the object the compiler produced for the same source, compiler, and flags, so the wheels do not change.
On some hosts the build container cannot resolve names through the runtime's default resolver, so pixi and git fail inside it unless each run is given a --dns workaround by hand. CONTAINER_DNS takes a space- or comma-separated list of servers and adds one --dns per entry to the container arguments, for both the dockcross runner (x64) and the direct run (aarch64). Unset, the arguments are unchanged. release_wheels.py runs each step with the caller's environment, so a CONTAINER_DNS exported by the user reaches the script. The module wheel script runs its container with --network=host, which uses the host resolver and which docker refuses to combine with --dns, so it is left unchanged.
The 30G limit was set only in the build's own environment, so any later ccache call outside the build (ccache -s, a manual run) saw the 5 GB default and could trim a warm cache to 5 GB on its next cleanup. Run `ccache --max-size <size>` before the build so max_size is written to the cache directory's ccache.conf. The size is the caller's CCACHE_MAXSIZE when set, otherwise the default; the builder passes it in so compiler_cache_session stays a pure function. sccache is unchanged: it has no per-cache-directory setting. Its disk cache size comes from SCCACHE_CACHE_SIZE or [cache.disk] size in the per-user config file named by SCCACHE_CONF (docs/Configuration.md), and it has no command to set a size.
Two release builds of the same ITK commit produced wheels whose every member had the same contents, but the wheel files differed because each zip entry carried the build time. scikit-build-core, wheel pack, auditwheel, delocate, and delvewheel all take their zip timestamps from SOURCE_DATE_EPOCH when it is set, so build_wheels.py now sets it to the committer time of the ITK revision being built. A value the caller already set is kept. A source tree without a repository (a tree restored from a build cache) has no commit time; the variable is then left unset with a warning rather than given an invented date, and git is stopped at the tree's parent so it cannot read an enclosing repository's time. The manylinux scripts forward a SOURCE_DATE_EPOCH set on the host into the container, as they do for the compiler cache settings.
build_wheels.py reads the ITK commit time inside each build. In the root-run aarch64 docker path git can refuse the mounted repository as "dubious ownership", which leaves SOURCE_DATE_EPOCH unset and the wheels' zip timestamps nondeterministic. release_wheels.py already holds the resolved ITK commit on the host, so it now reads that commit's committer time there and adds it to every step's environment, native and container. A value already set by the caller wins, through the shared build_wheels.source_date_epoch helper. The container scripts forward the host value into the container.
The cold Windows release build ended with `sccache --show-stats` reporting 0 compile requests although the cache grew. The server most likely idled out (SCCACHE_IDLE_TIMEOUT, default 600 s) during a long stretch without compiles, a later compile started a fresh server, and the final statistics came from that one. Set SCCACHE_IDLE_TIMEOUT=0 for sccache unless the caller set it. sccache 0.19 reads it when the server starts (src/server.rs, get_idle_timeout; 0 disables idle shutdown), and the session restarts the server after the environment is set and stops it at the end.
The step report resumes a run by skipping every step already marked done, whatever the run is building. A report from an earlier build of a different ITK commit therefore makes a new run skip all of its steps. BuildManager now takes an optional mapping of inputs, records it in the report, and starts every step again as pending when the recorded inputs differ or are missing. Identical inputs keep the resume behavior that a rerun after a failed step relies on. changed_inputs names the inputs that differed, so a caller can say why the steps run again.
Release build roots are fixed, so a second release-wheels run on a host reads the step report left by the first. Every step was already done, so a run for a new ITK commit skipped the ITK build and the wheel build and reported the earlier run's wheels as its own. The builder now records three inputs in the report: the commit checked out in the ITK source (resolved, since a branch name builds a new commit each day), the package version, and a fingerprint. The fingerprint hashes pixi.lock, the ITKPythonPackage scripts and CMake files, the builder's CMake definitions (which carry the compiler cache launcher), the compiler cache switch, and the manylinux version and image. Any change runs every step again in the kept tree and says which input changed; identical inputs still resume after a failed step.
dist/ is never emptied and release build roots are fixed. The macOS repair step runs delocate on every dist/*.whl, the import test installs itk with pip -f dist, and the release summary lists every matching wheel, so wheels left by an earlier run were repaired, tested, and reported as the new run's. Before the wheel step builds, the wheels at the top of dist/ now move to dist/earlier-wheels/<YYYYMMDD_HHMMSS>/. Nothing is deleted, and the globs and pip -f do not look into subdirectories. The move happens only when the wheel step runs, so a rerun with identical inputs that resumes after the wheel step keeps the wheels it built. The release summary's listing is a function now, so a test can show it skips moved wheels.
Two Windows builds of the same source gave wheels whose .pyd and .dll files, tbb12.dll among them, differed only in the PE COFF TimeDateStamp and the debug-directory TimeDateStamp: link.exe writes the link time there. SOURCE_DATE_EPOCH already fixes the zip timestamps, so these fields were all that kept the wheels from being identical. /Brepro makes the linker write a hash of the binary there instead. build_wheels.py now adds /Brepro to the _LINK_ environment variable on Windows, next to where it sets SOURCE_DATE_EPOCH, keeping any value the caller set and not adding it twice. Microsoft's "MSVC linker reference" (learn.microsoft.com/en-us/cpp/build/reference/linking, section "LINK environment variables") states that the linker "appends the options and arguments defined in the _LINK_ environment variable to the command line arguments before processing". link.exe reads it itself, so it reaches every link that CMake and Ninja run for ITK, the superbuild's ExternalProjects, the wheels, and remote modules, with no CMake flag changes and with CMake's default linker flags left intact. The windows-build environment uses the conda-forge vs2022_win-64 toolchain, which is MSVC link.exe. Linux and macOS are unchanged.
Replace the black hook with ruff-format, run after ruff-check, and drop [tool.black]. ruff also formats code examples in docstrings and Markdown, so blacken-docs is limited to the reStructuredText docs. The reformatted lines are f-string spacing and quotes in scripts/ and the quotes in one docstring example.
Replaces the shell and PowerShell wheel-build system with a pixi-driven
Python one, and moves ITK 6 to stable-ABI (
abi3) wheels onmanylinux_2_28, macOS arm64 14.0, and Windows x86_64. Supersedes #302.
Linux and macOS wheels now build with
--dead-stripby default (port ofSimpleITK/SimpleITK#2784):
itk_coreshrinks from 89.5 to 71.4 MB on Linuxand from 61.5 to 49.3 MB on macOS, with unchanged outputs.
Successfully built on Windows, macOS arm64, Linux x86_64, and Linux
aarch64. The Doxygen/environment defect found by @dzenanz is fixed: build
tools now resolve inside the target
--platform-env, and launching fromthe wrong environment fails immediately with the environment name to use.
Build caches for all four platforms are published to ITKPythonBuilds
v6.0rc01.dev20260915; the macOS cache targets macOS 14.0. Those cachesand the platform builds below predate the eighth package,
itk-filtering-extra; the current tree has been built only on Linuxx86_64 (all 8 packages, see the
--dead-stripmeasurements).Release decisions for reviewers: a new PyPI package,
itk-filtering-extra(the filtering modules ingested from remote modules), and
itk-ionowrequires
itk-numerics.Platform verification — clean builds from this branch
Every row is a from-scratch build of this branch, not an incremental one,
made before
itk-filtering-extrawas added (hence 7 wheels).cp311-abi3-manylinux_2_28_x86_64cp311-abi3-manylinux_2_28_aarch64cp311-abi3-macosx_14_0_arm64cp311-abi3-win_amd64macOS binaries were checked with
otool -l: real.sofiles reportminos 14.0, matching theirmacosx_14_0_arm64tag. That check mattersbecause a component wheel tagged below its true
minosinstalls on oldermacOS and then fails in dyld at import — installable and broken, rather
than loudly uninstallable.
Remote-module end-to-end (the consumer path)
Proven on Windows x86_64 and Linux x86_64, each run as a fresh
consumer with zero ITK wheels present beforehand, so a leftover
artifact could not produce a false pass.
Linux x86_64, ITKBioCell, 8 minutes,
rc=0:itk_biocell-1.0.0-cp311-abi3-manylinux_2_28_x86_64.whl, 0.58 MBRequires-Dist: itk<7,>=6.0.0b1— the dependency rewrite firedauditwheelpyproject.tomlrestored byte-identical after the buildAn earlier Windows pass was found to be riding on leftover wheels; the
zero-wheel precondition was added in response and both platforms were
re-run under it.
--dead-strip: measurements
Linux links with
-ffunction-sections -fdata-sections,--gc-sections,and a version script that exports only
PyInit_*from each Python module.--no-dead-stripopts out. User-Dflags such as-DCMAKE_CXX_FLAGS:STRING=...are merged with these, not replaced.--dead-strip_ITKCommonPythonexported symbolsitk_corewheel, Linux x86_64The export limit carries the saving;
--gc-sectionsalone gave 3-5%.Linux figures are from the pushed configuration (default linker, with a
user
-DCMAKE_CXX_FLAGS:STRING=-pipemerged in). Linking with gold gavewheels about 1% smaller (
itk_core70.7 MB) and was dropped. A remotemodule (ITKSimpleITKFilters) built against the stripped wheels gives
identical output in all four old/new ITK and module combinations.
Windows needs no change: a
.pydalready exports onlyPyInit_*(84 on_ITKCommonPython), and Release links use/OPT:REF,ICFby default.macOS Python modules link with
-dead_stripand export only_PyInit_*:wheels are 20-48% smaller (
itk_core61.5 to 49.3 MB), benchmark outputsare identical, and BSpline LBFGSB registration runs 15% faster.
-dead_stripalone changed nothing (<0.1%).The 8 ITK test failures occur with and without the change and come from
the wheel configuration (unwrapped return types such as
SinusoidSpatialFunctionF4PD4,TransformToStrainFilterforTransform[D,3,3],itk.Ransac) or missing test data(
Real-DSA-01-Speed-02.mhd).Unit tests and hooks
pytest: 205 passed. pytest is a developer-only dependency; thenormal build path does not import it.
pre-commit run --all-files: exit 0, every hookPassed(black, ruff, shellcheck, shfmt, taplo, yaml/toml, whitespace).
Known caveat: intermittent aarch64 link truncation
One clean aarch64 build failed while linking:
The identical clean build then passed, with no truncation and no short
object files. Evidence points at the container file-sharing layer rather
than this branch: 158 GB free at the time, the truncated archive was
exactly 49,152 bytes (12 × 4096, a dropped write tail rather than a
partial compile), and aarch64 is the only platform writing through a
Docker bind mount onto a macOS filesystem.
Docker Desktop updated itself between the two runs (29.7.2 → 29.8.0), so
the version cannot be credited with the fix, and one pass does not
disprove an intermittent fault. Recommended handling: retry a clean
build first; do not treat a truncation as a defect in this branch.
Commit series
Topical commits, no fixups and no merge commits, now including
@dzenanz's
wheels-6.0b3series and the--dead-stripcommits. SomeBUG:commits fix earlier commits in this series; they are kept separatebecause they record distinct defects found in testing.
Beyond the build-system replacement, the series also adds:
release-wheels, one command per release build machineitk-filtering-extra, a new package for the ingested filtering modulesitk-iodepending onitk-numerics--module-itk-dir install)ITK_BUILD_JOBSoverrides)Cavan Riley is credited via
Co-Authored-Byon the seven commitscarrying his original work.